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ABSTRACT 



North Carolina’s Schools Attuned program offers teachers an 



instructional approach that helps increase students’ classroom success by 
assessing strengths and weaknesses that influence learning. Approximately 400 
teachers participated in the program’s first year, 1998-1999, and an 
additional 400 began training in 1999-2000. Teams of teachers from a school 
receive intensive training in the summer, followed by monthly meetings during 
the school year. In the first program year, the evaluation team worked 
closely with program developers to refine the evaluation design and determine 
measurement strategies. During the second implementation year, evaluators are 
continuing to document the implementation process while placing more emphasis 
on outcome evaluation. Evaluators have been asked to expand the strategy as 
Schools Attuned begins operating as a national program. Throughout the 
evaluation, the collaboration between evaluators and program developers has 
heightened the evaluation effort. The initial evaluation design benefitted 
from the input of program developers, and evaluation information shared with 
program developers allowed for adjustments in program implementation. 
Evaluators and program developers emerged from the collaboration better able 
to assess the effects of the Schools Attuned Program. (SLD) 
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Collaborative Evaluation of Schools Attuned 



Rita G. O'Sullivan &. Bethany Page 
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School of Education 

University of North Carolina at Chapel Hill 27599 
919/843-7878 
ritao@emai Lunc.edu 



Overview 

The Schools Attuned program offers teachers on instructional approach that helps them 
increase students' classroom success by assessing strengths and weaknesses that influence 
learning. Beginning with the 1998-99 academic year. Schools Attuned received support from the 
North Carolina legislature to implement the program in 42 different schools across the state. 
Approximately 400 teachers participated in the program that first year and another 400 teachers 
began training in 1999-2000 with a third cohort to begin in Summer 2000. Teams of teachers from 
participating schools receive intensive training in the summer, followed by monthly meetings during 
the school year which are intended to assist teachers with understanding the approach and 
implementing it in their classrooms. 

During the first year of implementation in North Carolina, the evaluation team worked 
closely with program developers to refine the evaluation design and determine measurement 
strategies. The emphasis in the first year was to document what occurred os well os to pilot-test 
outcome evaluation approaches. During this the second implementation year, evaluators continue to 
document the implementation process while placing more emphasis on outcome evaluation. 
Additionally, the evaluators hove been asked to expand the evoloution strategies to the notional 
Schools Attuned program, which this summer will begin training teachers at nine sites throughout 
the United States. 

Throughout the evaluation, the intentionally collaborative approach between evaluators and program 
developers has heightened the evaluation effort. The initial evaluation design benefited greatly 
from the input of program developers. Evaluation information shared in o timely manner with 
program developers allowed for needed adjustments in program implementation. Development of 
certain outcome instruments required concerted collaboration. Together evaluators and program 
developers have emerged from the collaborative process better able to assess the effects of the 
Schools Attuned program implementation. 
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Background 

Dr. Mel Levine, drawing on more than 20 years as a pediatrician, developed a model of 
components (neurodevelopmental constructs) important to leorning that would help families and 
clinicians better understand learning difficulties. In 1987, soon after coming to the Center for 
Developmental Learning at the University of North Carolina at Chapel Hill, Dr. Levine was awarded o 
1.2 million grant to develop and disseminate o comprehensive professional development program for 
use with teacher in schools and named it Schools Attuned. Since then more than 5,000 teachers (in 
38 states os well as Canada and Denmark) hove been trained to use Schools Attuned techniques in 
their classrooms. In 1998, the North Carolina legislature approved the dissemination of Schools 
Attuned throughout the state. Critical to the North Carolina Schools Attuned initiative is 
evaluation. Dr. Rita O'Sullivan and members of her evaluation team were engaged to evaluate the 
North Carolina Schools Attuned effort, thereby developing prototypical evaluation strategies for 
future endeavors. 

The Schools Attuned model, as developed by Dr. Mel Levine, is clinically grounded and 
supported by extensive research literature. Each of the neurodevelopmental functions within the 
eight constructs of the model are commonly written about and observed by clinicians, educators, 
and researchers as important to students' academic success. Thus the validity of the 
neurodevelopmental constructs has been adequately established. Beyond construct validation of 
the model, however, and more important to All Kinds of Minds is validation of the model's 
application. This is to say, "Does using the model actually result in improved achievement for 
students?" Systematic evaluation of the model's application should be the key issue for Schools 
Attuned. 



For the past 10 years. Schools Attuned has survived and continues to grow, which nteans 
that its face validity among practitioners has been soundly established with an additional 800 
participants expected to begin training in Summer 2000. Further, anecdotal evidence abounds about 
the positive effects of the program for children, including the WSBH television series to be 
broadcast in January 2001. Data are emerging from the evaluation of the North Carolina Schools 
Attuned program, begun in 1998, that suggest positive teacher and student outcomes. Given 
projected expansion of the National Schools Attuned program, continued, broader-based, and more 
formal evidence of program effectiveness is certainly warranted. 



• Evaluation Frameivork 

Schools Attuned is an instructional approach that assumes changes in teachers' (or any 
educators') understanding of neurodevelopmental constructs and their functions will allow them to 
work more successfully with students who experience difficulty learning. Based on this new 
understanding, the teachers are expected to implement changes in their classroonw that should 
result in a number of positive students outcomes, including improved achievement and other 
classroom success factors. The most common sequence of events for effective instructional 
programs is: 

Teocher Training — ►Teacher Understanding ► Classroom Changes ► Student Success 

From an evaluation viewpoint, each of the events must be evaluated in sequence. A poor training 
program is unlikely to produce positive changes in teacher understanding. If teachers' 
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understanding does not change, classroom modifications ore unlikely. Similarly, the types of 
changes made in the classroom will influence the desired final outcome of improving students' 
classroom success. This is particularly true with programs that evolve and improve over time. As 
programs emerge, evaluation of training and of participants' understanding that results from 
training (formative evaluation) is extremely important. As programs become more established and 
the quality of the training assured, then evaluation of outcomes (summative evaluation) becomes 
more important. 



Evaluation Findings 

North Carolina Schools Attuned has afforded programmatic and evaluative opportunities for 
All Kinds of Minds. Two summers have yielded approximately 800 teachers and educators from 
almost 20 different school districts in the state who are participating in the program. State 
support for Schools Attuned has created a laboratory for evaluation that can be exported with 
some modifications to the National Schools Attuned efforts. With North Carolina Schools 
Attuned, we have followed the evaluation framework described above that looks at the sequence of 
Teacher Training to Teacher Understanding to Classroom Changes to Student Success. We have 
emphasized formative evaluation during the first year so that we could provide feedback to the 
program developers that would allow them to modify the program as needed. We also have 
developed a number of evaluation strategies and tools so that we are poised to begin a more 
summative evaluation of the project this year. 



TEACHER TRAININ6 



Where We Are 


What We Know 


What We Are 
Learning 


* Evaluation of Troining has been a strong evoluation 
component for some time. For North Coro lino 
Schools Attuned we odded o mid-year PorticiDont 
Phone Interview to determine how teachers and 
other school stuff in the program viewed the quality 
of the octivities and their participation 


Participants are generally positive 
obout their participation. This 
evoluative information did trigger 
some refinements to the program. 


Ways program 
developers 
continue to 
improve the 
program 



TEACHERS' UNDERSTANMNS 



Where We Are 


What We Know 


What We Are 
Learning 


• Building on two complex case studies used in training, 
evaluators and program stuff developed scoring 
criterio to compare participants' enhanced obility to 
use neurodevelopmental constructs. 


Refinement of the scoring process 
continues. Case studies were 
completed by oil 1999-00 
participants during summer 1999 
training. 


Extent that 
participants in the 
program 
understand 
Schools Attuned 
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OASSROOM CHANGES 



Where We Are 


What We Know 


What We Are 
Learning 


• Chances in Teocher Behaviors can now be compared 
using a pre> and post- survey that asks teacher how 
they respond to students with various learning 
difficulties. 


Teachers report relying less on 
outside assistance (e.g., special 
education teachers, counselors, 
etc) and more on instructional 
strategies (e.g., individualizing 
assignments, demystification, etc.) 
with students experiencing 
difficulty. 


How teachers ore 
changing 
strategies that 
they use in their 
classrooms to help 
students succeed 


• Demonstratina Schools Attuned in Action was pilot 
tested by video-taping 4 teachers during a half-hour 
lesson and then having them explain how what they 
did In the classroom is related to their 
understanding of Schools Attuned. Students' 
understanding of the program also wos assessed 
through focus groups and drawing. 


Additional pilot testing is needed 
for this promising source of rich 
evaluation information 


Once teachers 
understand 
Schools Attuned, 
extent that they 
change their 
classroom 
practices 


• Chances in Classroom Clinwrte is tracked usinq a 
survey of students, specifically developed for 
Schools Attuned, that was successfully pilot- tested 
during spring 1999. For the 1999-CX) school year, 
students in all North Carolina Schools Attuned 
classrooms will complete a pre- and post- assessment 
of their classroom environment. 


To measure the validity of the 
survey, a factor analysis was 
conducted and confirmed that the 
classroom climate survey had 
construct validity. Student results 
will be available summer 2CXX) 


Students 

perceived changes 
in their classroom 
environment, 
relative to School 
Attuned 



STUDENT SUCCESS 



Where We Are 


What We Know 


What We Are 
Learning 


1999*2000 teachers will complete end-of*year 
surveys that will share changes in students 
standardized test scores in relation to teachers' 
use of Schools Attuned with these children 


Results should be available by 
Summer 2000 


Whether student 
achievement 
improves as a 
result of Schools 
Attuned 


• For a sample of students in North Carolina. 
Standardized test scores for students in 
porticipants' classrooms will be compared with their 
previous year's scores. 


Wake and Durham, NC school 
districts are making scores 
available and analyses should be 
completed during 2000 




• A self-concept measure was pilot tested in 4 

classrooms to nvike sure it was appropriate for the 
age groups. For the 1999-CX) school year, students in 
all North Carolina Schools Attuned classrooms will 
complete a pre- and post- assessment of their self- 
concept. 


Results from the pilot testing were 
positive. 


Whether other 
student outcomes. 
Including self* 
concept. Improve 
as a result of 
Schools Attuned? 
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Recommendations for All Kinds of Minds Evaluation Efforts - Year 2000 

After 18 months, the North Carolina Schools Attuned evaluation is well on its way to 
gathering more evidence to answer the "What We Need to Know* questions. Unlike research that 
occurs in controlled environments, evaluation of programs in schools settings is an approximate 
science at best or rather a series of progressively more convincing approximations. Case-control 
methodology is resource intensive and not terribly useful with school events. More often than not, 
the time, money, and energy expended in identifying and maintaining control groups yields 
unsatisfactory findings. Rather, it is internally more valid to have students and teachers os their 
own controls in pre-test/post-test situations. Similarly, replications of positive findings (across 
participants and across sites) build extremely convincing evidence of program effectiveness. If 
Schools Attuned can demonstrate that students in participant North Carolina classrooms show 
greater than average academic growth for their school district and the same can be said for 
students in New Jersey or New Mexico, then the cumulative evidence becomes progressively 
weightier. Based on these considerations, I make the following recommendations for evaluation: 

• Focus on the national implementation of Schools Attuned. 

• Expand the North Carolina collaborative evaluation approach so that external evaluators work with 
program developers to integrate evaluation into the national Schools Attuned programs. 

• Adapt instruments and evaluation practices developed for North Carolina Schools Attuned for the 
national program. 

• Continue to support evaluation activities that will gather successively convincing evidence to convey 
detailed stories of the effectiveness of School Attuned. 

Evaluation Plan for Notional Schools Attuned 

The National Schools Attuned program should be the central, immediate focus for the All 
Kinds of Minds evaluation. The North Carolina Schools Attuned program has had 350-400 
participants each year with teams of teachers participating from the same school. Given these 
numbers and a service delivery system that has multiple teachers at one school, the North Carolina 
Schools Attuned evaluation should continue to provide a laboratory for instrument development and 
for testing evaluation methodologies. This said, an evaluation focus on the National Schools 
Attuned program would encompass: 1) modifying and integrating proven evaluation methods into the 
National Schools Attuned curriculum, 2) developing uniform and user-friendly mechanisms so that 
evaluation information is summarized and shared with appropriate stakeholders in a time and cost 
effective manner, 3) strategically designing studies, conducted by the external evaluation team, 
that will permit the collection of progressively convincing evidence of the effectiveness of Schools 
Attuned. 



Importance of Working Colloborotively 

Evaluation of the National Schools attuned program should occur through collaboration 
among program evaluators, developers, trainers, and implementers. To develop a useful, cost 
effective evaluation approach all stakeholders need to be actively engaged in the evaluation efforts. 
Just as evaluators strive to collect and make sense of evidence across national training center 
sites, trainers need to make sense of evidence across the many teachers they train, and teachers 
need to make sense of their observations of student behaviors. All of these are evaluation 
activities, and they need to be carefully developed. 
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Plon of Evoluotion Activities 

Winter 2000 

• Develop detailed evaluation plan with input from National Schools Attuned faculty 

• Review All Kinds of Minds Documents relevant to National Schools Attuned program 

• Contact Training Centers to determine current practices and gather suggestions for the evaluation 

• Examine North Carolina Schoals Attuned Evaluation results for adaptation to National Schools Attuned 

Spring 2000 

• Work with National Schools Attuned program staff to identify opportunities to imbed evaluation into 
curriculum practices. 

• Select 4 national training center sites for cross-site studies 

• Prepare evaluation instruments for summer training activities 

• Present evaluation design to National Training Faculty for discussion during scheduled training 

• Modify evaluation plans os needed based on National Faculty review 

Summer 2000 

• Distribute evaluation instruments to National Faculty 

• Determine data collection and data summary needs for each National Training Site 

• Observe Schools Attuned Training at National Training Centers 

• Develop evaluation plans for each of the National Training Sites 

Fall 2000 

• Distribute students climate and self-concept instruments to National Training sites as reguested. 

• Summarize data os reguested for National Training sites 

• Conduct site visit to National Training sites and interview selected Schools Attuned participants 

• Provide technical assistance in evaluation as needed to National Training Sites 

Winter 2001 

• Review and revise evaluation plan with input from National Schools Attuned faculty 

• Review All Kinds of Minds Documents relevant to National Schools Attuned program 

• Contact Regional Training Centers to evaluate mid-year implementation 

• Re-examine North Carolina Schools Attuned Evaluation results for adaptation to National Schools 
Attuned 

Spring 2001 

• Continue to work with National Schools Attuned program staff to identify additional opportunities to 
imbed evaluation into curriculum practices. 

• Revise evaluation instruments as needed for summer training activities 

• Present National Schools Attuned evaluation design to National Training Faculty for discussion during 
scheduled training 

• Modify National Schools Attuned evaluation plan os needed based on National Faculty review 
Summer 2001 

• Distribute evaluation instruments to National Faculty 

• Hold Schools Attuned Evaluation Showcase to share national and site evaluation results 

• Revise as needed data collection and data summary needs for each National Training Site 

• Observe Schools Attuned Training at National Training Centers 

• Revise evaluation plans for each of the National Training Sites 

• Collect available student achievement data for 2000-2001 academic year 
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